Incorporating Gesture and Gaze into Multimodal Models of Human-to-Human Communication

In human communication, utterances are expressed with some structural events, e.g., sentence, speech repairs, control of floor, and etc. These structural events bring important information and quite helpful for a better understanding of human communication. Meanwhile, the human communication is also full of multimodal behaviors, e.g., gesture, gaze, and etc. As non-verbal signals, gesture and gaze show closed temporal and semantic links to spoken content. In my thesis, I am working on incorporating non-verbal cues in a multimodal model to better predict the structural events to further improve the understanding of human communication. Some research efforts made are summarized and my future research plan is described.